Categorical Foundations of Gradient-Based Learning

Abstract We propose a categorical semantics of gradient-based machine learning algorithms in terms lenses, parametric maps, and reverse derivative categories. This foundation provides powerful explanatory unifying framework: it encompasses variety gradient descent such as ADAM, AdaGrad, Nesterov momentum, well loss functions MSE Softmax cross-entropy, shedding new light on their similarities differences. Our approach to has examples generalising beyond the familiar continuous domains (modelled categories smooth maps) can be realized discrete setting boolean circuits. Finally, we demonstrate practical significance our framework with an implementation Python.